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ABSTRACT 

We describe the software requirement and design specifications for all-sky panoramic 
astronomical pipelines. The described software aims to meet the specific needs of 
super-wide angle optics, and includes cosmic-ray hit rejection, image compression, 
star recognition, sky opacity analysis, transient detection and a web server allowing 
access to real-time and archived data. The presented software is being regularly used 
for the pipeline processing of 11 all-sky cameras located in some of the world's premier 
observatories. We encourage all-sky camera operators to use our software and/or our 
hosting services and become part of the global Night Sky Live network. 

Key words: Methods: data analysis Techniques: image processing. 



1 INTRODUCTION 

In the past several years, all-sky cameras have be- 
come increasingly popular in the field of astronomy, 
and many systems incorporating super-wide fish-eye 
optics have been developed, installed and operated. 
Applic ations of all-sky cameras in c lude cl oud detec- 
tion jHull. Limmongk pl fc, Siegmundl Il994l : iHogg et alJ 
1200 ll: iTakato et alJ l2003h bright star monitoring 
JShamir fc Nemiroffl |2004 K 
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1998; Bro sch fc Man ulis 2002; S purnv. Oberst fc Heinleinl 
2003). However, while a substantial portion of the effort 
is directed toward the design of the hardware, less has 
yet been reported on the development of the software 
required for the autonomous operation of all-sky cameras. 
This software development c an sometime s turn into an 
unexpectedly demanding task <lBrooksl|l987l) and may delay 
the beginning of the effective operation of the camera. 

In this paper we describe a software system that enables 
the operation and the analysis of astronomical pipelines. The 
algorithms, software and hosting services described in the 
paper can be used by all-sky camera operators who seek to 
increase the effectiveness of the data produced by their cam- 
eras, yet prefer to avoid the costs of software development. In 
Section |21 we describe the design of the software running on 
the remote station, including cosmic-ray hit rejection, star 
recognition, opacity maps and transient detection, and in 
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Section[3]we describe the software design of the main server, 
which collects and archives the data transmitted from the 
remote stations. 



2 REMOTE STATION SOFTWARE DESIGN 

The software that runs on the remote station is based on 
Linux Red-Hat, which is considered robust comparing to 
other existing PC operating systems. The remote station 
starts taking exposures in time intervals of 236 seconds. For 
instance, if the duration of the exposure is set to 180 seconds, 
an image will be taken every 236 seconds. If the duration is 
set to 360 seconds, an image will be taken every 472 seconds. 
Since 236 second intervals are used, a given image can be 
compared with another image taken on a different day, but 
at the exact same sidereal time so that the positions of the 
stars appearing in the two images are identical. For instance, 
the positions of stars recorded by an image taken at 12:00:00 
UT are the same as the positions of stars recorded on the 
previous day at 12:03:56 UT or at 12:07:52 two days before. 
This feature is used for all-sky opacity maps and transient 
detection, which will be described later in the paper. 

Images are taken only after sunset (based on USNO 
data) and before dawn. No images are taken during the day. 
The system also changes the duration of the exposure ac- 
cording to the position and phase of the moon. When the 
moon is up, the duration of the exposure is shorter. 

In order to support as many stations as possible, yet 
without overloading the main server, all CPU-intensive im- 
age processing is done on the computer hosting the camera 
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(the remote station). This approach takes advantage of the 
free CPU resources available while the camera is taking an 
exposure. 

2.1 Cosmic ray-hit rejection 

After each new exposure is taken, cosmic ray-hits are 
rejected from the frame so the image can be pro- 
cessed. One technique of cosmic-ray hit rejection 
is by comparing several exposure s of t he same field 
i Windhorst. Franklin fc Neuschaeferl Il994l : iFixsen et alJ 
2000). However, since one of the main purposes of the de- 
scribed system is to automatica lly detect optical transients 
that appear in singl e im ages dShamir fc Nemirofj EoOodh 
as well as meteors JShamid l2005blL such techniques are 
not suitable for this system. In order to reject cosmic-ray 
hits without rejecting true transients, a fuzzy-logic based 
algorith m for c osmic-ray hit rejection from single images 
( Sham!] l2005al) is applied. Like some other cosmic-ray 
hit rejection algorithms from single images, the rejections 
make use of the non-point source nature of cosmic ray 
splashes. Even though this algorithm does not introduce 
any better accuracy than other reported cosmic-ray hit 
rejection algorithms, it has a clear advantage in terms of 
computational complexity, which is a major concern in 
high image-rate pipelines. While some of the previously 
propo sed algorithms may be too slow for systems a t this 
scale (iRhoadsl Il99l IVan Dokkuml 1200 it |Pvchll2004) . the 
fuzzy-logic based algorithm proc esses a 1024x1024 frame 
in just 12 seconds |Shamirll2005a). 

2.2 Star recognition 

In order to monitor bright stars, the software associates stel- 
lar objects visible in the frames with known-catalogued ob- 
jects. This is done by applying a fuzzy-logic based star recog- 
nition algorithm desi gned specifically for super -wide angle 
astronomical images dShamir fc NemirofjE)05ah . 

For each recognized star, the intensity of the local back- 
ground is estimated and photometric data are collected. The 
intensity of the local background is estimated by the median 
of the 1600 pixels surrounding the point spread function 
(PSF). The intensity of the star is estimated by averaging 
the brightest pixels of the stars PSF and then subtract- 
ing the estimated local background. For each bright star 
(brighter than a certain preset magnitude), the averages of 
the brightest 1, 5, 9, 16 and 25 pixels are collected. For dim- 
mer stars, however, the realized PSF is smaller so only the 
average of the top five pixels is currently listed. The average 
of the brightest pixels corresponds to their sum, but provides 
smaller numbers that can be handled using 16-bit integers. 
This reduces the bandwidth and storage space required to 
store these data. 

Photometry files also list for each detected object some 
additional data taken from the Henry Draper catalogue such 
as the visual magnitude, the spectral type and celestial co- 
ordinates. The data are then stored in text-based HTML 
files that can be viewed by internet browsers and elec- 
tronic spread sheets. The HTML files are xml-tagged, so 
that customized external software tools can easily browse 
and collect the data. The photometric precision of a sin- 
gle measurement is around 0.2 magnitudes, which allows 



star variability monitoring of high-amplitude va riable stars 
such as Algol (iMuzzin. Shamir fc Nemirofl1l2005l) . However, 
using measurements obtained over several nights can lead 
to photometric precision of 0.02 magnitudes, which allows 
monitoring low-ampl itude variable stars such as Polaris 
jNemiroff et alJl2005l) . 

Another product of this processing is the generation 
of annotated JPG images, whic h labels the names of th e 
constellations and bright stars dShamir fc Nemirofj I2004T) . 
These images can be used for "cosmetic" purposes, but can 
also be ap plied to educati onal purposes such as studying the 
night sky (iNemiroffeOOlil) . 

2.3 Opacity maps 

All-sky cameras can be used as simple cloud mo nitoring 
devices by simply inspecting the images by eye llRuttenl 
120031: IPereira et al]l2005l . 120061) . However, while thick clouds 
can be easily noticed, thin cirrus clouds at night are often 
invisible to the unarmed human eye. In order to provide 
a more informative view of the opacity of the night sky, 
photometric measurements of numerous background stars 
are combined with simultaneous sky brightness measure- 
ments to differentiate thin cloud s from sky glow sources such 
as air glow and zodiacal light dShamir fc Nemirofll f2005b; 
iNemiroff fc Shamirl EooH) . This is done by comparing all- 
sky images to canonical images taken on other nights at the 
same sidereal time. The canonical images are collected au- 
tomatically by counting the number of stars brighter than a 
certain magnitude detected by the star detection algorithm 
JShamir fc NemirofdE)05ah . If the PSFs of a sufficient per- 
centage of expected stars are found in the image, the image 
is classified as clear, and added to the database of canonical 
images where it is co-added to all other clear images taken 
at the same sidereal time. 

When an image is taken, any PSF brighter than its 
background by a given a is assigned with two values: a stellar 
intensity, and an estimated intensity of the background, as 
explained in Section 12.21 The values assigned to the PSFs 
are then compared to the values of the PSFs of a canonical 
image. The comparison is based on Equations and [5] 

M-star — T • Istar T ■ Ispace ~\~ Icloud (1) 
M ha + Icloud (2) 

where M s tar is the intensity of the light coming 
from pixels at the location of a star in the given image, 
Mbackground is the intensity of the light coming from pixels 
not at the location of a star, htar is the intensity of the star, 
/space is the intensity of light from background space, I c ioud 
is the intensity of the cloud covering the star and T is the 
transmission around the star. Using Equations and [5] the 
transmission is determined by Equation [3] 

„ _ (M Bt ar - M ha 

ckground) ($) 

Istar 

where I s tar is based on measurements taken from the 
canonical image such that I sta r = Mo star - Mo background , 
where Mo sta r and Mobackground are the measured intensity 
of the star and the background, respectively, in the canonical 
image. This calculation of Istar is based on the assumption 
that no light is lost as starlight travels through air on a clear 



Software design for panoramic astronomical pipeline processing 3 



night. This assumption is not true, however, and provides 
a simplification of the problem that allows estimating the 
normalized transmission comparing to a clear night. This 
simplification gives 

j, (P^fstar ^"[background) {^) 

(MOstar — MOb ackground j 

Since the desired product is a broad relative transmis- 
sion map, the transmission of each pixel is calculated by 
interpolating the transmission measured directly along the 
lines to the stars in the image. This is performed by choosing, 
for example, the four nearest stars Suft, Stop, S r i g ht, Sbottom 
such that \Y S i - Vo| < X - X SI , \Y Sr - Y \ < X Sr - X , 
\X St - X \ < Y - Y st , \X Sb - X \ < Y Sb - Y , where 
Xsn is the X image coordinate of the star S n , Ys n is the 
Y image coordinate of the star S n , and (Xo,Yo) are the 
image coordinates of the given pixel whose sky transmis- 
sion is being estimated. After finding the four nearest stars, 
a two-dimensional linear interpolation of the transmission 
measured by the four stars is performed, and the calculated 
value is determined as the relative transmission of the pixel. 
The computed relative transmission values of all pixels in 
the frame are then used for generating the all-sky opacity 
maps such that a high relative transmission implies high rel- 
ative sky opacity and low relative transmission implies low 
relative opacity. 

The computed normalized transmission T provided by 
this method is not intended to be used for scientific pur- 
poses such as photometric reduction. However, providing a 
numeric indication of the estimated atmospheric transmis- 
sion is believed to have an advantage over a simple boolean 
clear/cloudy bit of information. The numeric indication can 
be used by robotic telescopes for the purpose of decision 
making (for instance, weighing the importance of the obser- 
vation against the atmospheric transmission) and can also 
provide human observers with more informative indications 
regarding the visibility conditions of the area of the sky they 
observe. 

In order to visualize the sky opacity map, the pixel 
is coloured in blue such that a stronger blue colour rep- 
resents higher opacity. The maps are generated by chang- 
ing the B component of each RGB triple such that B — 
BqT + 255(1 — T), where Bo is the original B value of the 
pixel and T is the sky transmission calculated using Equa- 
tion^ A scale added to the top left part of the transmission 
map presents the colours of several levels of opacity such 
that AM = — 2.51og 10 T. Currently, opacity maps are eval- 
uated visually by human observers, but the data may also be 
used to improve the effectiveness of robotic telescopes and 
autonomous sky surveys. Figure is an example of an all- 
sky opacity map generated by the all-sky CONCAM camera 
operating in Cerro Pachon, Chi le. A detailed description o f 
the opacity maps is available at dShamir fc Nemirofll2005bl) . 



2.4 Image compression 

Some of the best sites for astronomical observations are lo- 
cated far from urban centers or other sources of light pol- 
lution. Therefore, a fast internet connection is not always 
available at some of these sites. Since FITS files consume 
a relatively large space, a data compression algorithm is 



sometimes required in order to maintain a high image-rate 
pipeline. 

The FITS image compression algorithm used 
in the design described in this paper is PHOTZIP 
jShamir fc Nemirofj|2005^ . PHOTZIP works by modeling, 
smoothing, and then compressing the astronomical back- 
ground behind self-detected point sources while completely 
preserving values in and around those sources. Since the 
compression of the many background pixels is done in a 
symmetric fashion, the statistical properties such as mean 
and median (assuming normal distribution) are practically 
preserved. Since the values of the bright pixels are also pre- 
served, photometry measurements of I s tar — Ibackground in 
the compressed/decompressed FITS images are practically 
identical to the original image. 

Another advantage of PHOTZIP is that it allows users 
to control the compression/lossiness trade-off in terms of a, 
so the compression factor can be adjusted to the available 
bandwidth at the site while users have an accurate sense of 
the lossiness due to the algorithm. 

As an example of an existing applic ation, the above al- 
gorit hm is used regularly in CONCAM jNemiroff fc R afcrt 
1999) all-sky camera operating at SALT observatory (South 
Africa) with a compression factor of 85%. Experiments 
show that the photometric measurements in the com- 
pressed/decompressed images are practically the same as in 
the original images. 

A more de tailed description of the PH OTZIP algorithm 
is available at (Sha mir fc Nem irofI 20053). 

2.5 Transient detection 

All-sky wide angle astronomical pipeli nes can be used for the 
purp ose of optical transient detection jKroll fc Fleischmannl 
2001). Since the ability of all-sky optics to observe faint 
sources is not comparable to even simple narrow-angle tele- 
scopes, only bright transients can be detected by all-sky 
super-wide angle devices. However, the low limiting mag- 
nitude of all-sky optics can be weighed against the large 
portion of the sky covered by the wide lens. 

Brighter astronomical transients are sometim es con- 
sidered more int eresting to science llLindlevI Il987t 
iMenzies et al .1198 It) . Naturally, a wide-angle system is much 
more likely to capture such transients than narrow-angle 
telescopes covering only a very small portion of the sky 
at any given moment. In fact, one can reasonably assume 
that some bright short timescale transients visible to the 
unarmed eye are n ot noticed at a ll by the available narrow- 
angle sky surveys <ISchaeferll2005T) . 

In the proposed system, all-sky tra nsient detection 
JShamirll2005d: IShamir fc NemirofllfroOSdl) requires several 
logical steps. The first is the rejection of pixels domi- 
nated by cosmic-ray generated counts, as described in Sec- 
tion 12.11 Next, bright planets and high-amplitude vari- 
able stars are also rejected using a star recognition algo- 
rithm d esigned to find astronomic al objects in wide angle 
frames JSh amir fc Nemiroffll2005aD . After cosmic-ray hits 
and bright planets are rejected, the system searches for PSFs 
that are a given a brighter than the local background. Each 
PSF is then compared with the corresponding PSF in the 
canonical frame taken from the database of canonical images 
described in Section ^. 31 The absence of moving parts in all- 
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Figure 1. All-sky opacity map generated in Cerro Pachon, Chile 



sky instruments simplifies the comparison of all-sky images 
so that finding corresponding PSFs in two images taken at 
the same sidereal time is relatively easy. The detected tran- 
sients are listed in an xml-tagged HTML file and transmitted 
to the main server where they are available in real time. The 
current version of the transient detection mechanism does 
not provide information regarding the shape of the tran- 
sient PSF so that users who wish to analyze the nature of 
the transient are required to observe the image by eye. 

This system is currently operating in two all-sky cam- 
eras located in La Palma, Canary Islands and Cerro Pachon, 



Chile, and searches for optical transients brighter than 20a 
over their local background. 

One down-side of this transi ent detection mechan ism is 
noise produced by satellite glints iSchaefer et al .1198 It) . This 
problem can be solved by comparing the detected transients 
with a database of predicted known satellites glints such as 
the one provided by Heavens-Above.com. This feature is not 
yet implemented in the current system. Experiments based 
on a device with a limiting stellar magnitude of 6.8 showed 
that approximately 5 bright transients are reported each 
night, most of which are satellite flashes and some others 
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are bright meteors. Procedures performed in the server-side 
of the system reject satellite glints by comparing data from 
different cameras and by looking for sources that seem to 
rotate with the sky, as will be described in Section |3~21 



3 SERVER SIDE SOFTWARE DESIGN 

The purpose of the server is to receive and archive the data 
sent from the nodes, generate GIF animations presenting 
the whole night as a movie, and provide web access to the 
data using a simple internet browser. 

3.1 Web server 

In order to increase the availability of the data, all files 
are stored on a web server and are made accessible via 
the internet. The web site is organized such that the front 
page is a world-map labeled with the locations of the ac- 
tive all-sky cameras. Clicking on a location leads to a web 
page showing the last image taken at that site. Each loca- 
tion has an archive that provides access to previous data. 
The archive provides the FITS images, opacity maps and 
annotated JPG images described in Section 12.21 In addi- 
tion, photometry files (also described in Section 12. 211 arc 
also provided. The files are xml tagged, so that external 
software tools can easily collect data from the server. The 
URL of each file is also of a standard format which is 
/xx/xxyymmdd/xxy ymmddUThhnnss.html, where xx is the 
symbol of the observatory, and yy, mm, dd, hh, nn, ss are 
the year, month, day, hour, minute and second of the begin- 
ning of the exposure, respectively. This file naming policy 
makes it simpler to find specific web pages, and also makes 
it easier for external automatic tools to navigate through 
the data. 

The data are also copied to a database which allows per- 
forming simple queries. The database is based on the open 
source MySQL. All queries are performed on-line using a 
web-based interface that supports queries according to stars 
(Henry Draper catalogue number), date and time. 

Once the last image for the night is transmitted to 
the server, a GIF movie, which is a sequence of all images 
recorded that night, is generated on the server and placed in 
the archive. We found this feature useful for quick viewing 
of entire nights. 

In addition, the web server also provides a discussion 
board where users can share their thoughts and experiences. 
The discussion board also serves as a log by documenting 
interesting events in a public fashion. 

3.2 Transient detection (server side) 

In Section l2.5l we described a mechanism that detects tran- 
sients in single images. The result of the single-image tran- 
sient detection generated in the remote sites is transmitted 
to the server in the form of xml-tagged files. Each file con- 
tains a list of transients detected so far that night, and each 
entry in the list includes the date, time, approximate celes- 
tial coordinates (RA,DEC) and the filename of the image 
in which the transient was detected. The RA, DEC coordi- 
nates are obtained by using the fuzzy-logic based coordinate 



transfo rmation algorithm described in iShamir fc Nemirofj 
l2005aft . 

A computer program running on the server reads the 
files transmitted by the remote stations and searches for 
transients detected in the same celestial coordinates and at 
the same time. A transient recorded by two separate all-sky 
cameras in the same region of the sky at the same time may 
indicate that the source is astronomical, and not a satellite 
or a bright meteor. 

Another criterion taken into account is persistency. The 
system searches for transients appearing at the same geocen- 
tric celestial coordinates for at least two consecutive expo- 
sures taken by the same camera. The purpose of this mech- 
anism is to find optical transients that rotate with the sky. 



3.3 Meteor science 

Networ ks of all-sky cameras are often used for meteor 
scien c e llSopedll99a lOberst et alJll99gl : iBrosch fc Manulisl 
2002; Spu rnv. Oberst fc Heinleinl 12003) and c an provide a 
comprehensive coverage of meteor showers iRafert et alJ 
1200 li iFleming et"ai1l2002l ; IBrosch fc Manulisl I2002T) . It also 
proved to be efficient in the search for sporadic fir eballs 
ilOberst et al.lll99d : Ispurnv. Oberst fc Heinleinl I2003D . The 
described software system allows users to access images 
recorded during meteor showers and automatically detects 
sporadic bright fireballs as discussed in Section [2.51 

Another feature related to meteor science using all-sky 
cameras is the 3D analysis of meteor trails by usin g data 
from t wo (or more) neighbouring all-sky cameras dShamirl 
l2005bl) . Since exposures at different stations start at the 
same time, it is likely that a meteor recorded by one camera 
is also recorded by another nearby camera. After transform- 
ing the image coordinates of th e start and end of the meteo r 
trail to celestial coordinates JShamir fc Nemirofj l2005ah . 
simple triangulation can provide the altitude of the start 
of the meteor trail, the altitude of the end of the meteor 
trail, and the a bsolute trails length with accuracy of ~3% 
JShamirll2005rl . When a meteor has more than two peaks 
of maximum brightness, th e absolute dista nce between each 
peak can also be obtained JShamirll2005rl . 



4 CONCLUSIONS 

We described a software design that aims to provide a com- 
prehensive software solution to all-sky panoramic astronom- 
ical pipelines. The software controls both the remote station, 
at which images are taken and processed, and the main 
server that archives and provides access to the data. The 
described software is currently used for the processing of all- 
sky panoramic astronomical pipelines generated by 11 all- 
sky cameras located in Mauna Kea and Haleakala (Hawaii) , 
Cerro Pachon (Chile), Kitt Peak (Arizona), Mt. Wilson 
(California), Rosemary Hill (Florida), Siding Spring (Aus- 
tralia), Wise Obs. (Israel), Canary Islands (Spain), SALT 
(South Africa) and Hanle (India). The data collected in 
these observatories are available at |http: //n ightskylive.net 
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